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ABSTRACT 

This quasi-experimental study was designed to explore the potential impact of online 
instruction in a graduate course affects K-12 teachers' attitudes towards technology 
integration in schools and learning performance in the United States. This study used a 
nonequivalent control group design. Nineteen participants in both the experimental group 
(online section) and control group (traditional section) were pretested and posttested with 
the Stages of Concerns (SoC) Questionnaire in the fall semester of 2003. Due to the 
unbalanced participants in the experimental and control groups, the nonparametric statistic 
procedure was used to examine the differences between two groups. The Mann-Whitney U 
test indicated a significant difference only existed in stage 2-personal stage in both raw 
score and percentile score between the online and traditional sections. That is, online 
instruction significantly promoted online learner's concern only in stage 2-personal stage. 
No significant differences were found in other six stages of SoC questionnaire between the 
two groups. In addition, no significant differences were found in students' final course 
grades between two groups. Implications for K-12 teacher education were proposed. 

Keywords: online instruction, quasi-experimental study, learning performance, no significant 
differences, significant differences, attitudes toward technology integration 

INTRODUCTION 

Online instruction generally refers to the fact that the student is separated from the teacher 
and connected through the use of a computer and the Internet. On the other hand, 
traditional classroom instruction generally refers to the face-to-face instruction, typically 
conducted in a classroom setting and primarily involving in a lecture/ note taking model. 
Currently, online instruction is a primary method for distance education. An increasing 
number of institutions are offering online courses and/or programs to their students in order 
to meet various learners' needs. According to Waits & Lewis (2003), distance education has 
grown fast in recent years. In the 2000-2001 academic year, 56% of all 2-year and 4-year 
institutions offered distance education courses for various learners. In addition, 12% of all 
institutions planned to offer distance education courses in the next 3 years. 

Recent research has indicated that online education has positively influenced many aspects 
of education both directly and indirectly (CEO Forum, 2000; Phipps & Merisotis, 1999). 
Previously there have been related debates in this area. Clark (1983, 1994) maintained that 
media do not influence learning in any condition. But Kozma (1994) debated that 
educational technologies will influence learning by interacting with an individual's cognitive 
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and social processes in constructing knowledge. These earlier debates are still relevant since 
newly emerging technologies allow users to use them more efficiently. Most recent research 
studies in this area have been conducted around these debates. 

According to Phipps and Merisotis (1999) and Russell (1999), there have been two lines of 
research regarding the effects of online vs. traditional instruction on students' end-of- 
semester grades or learning outcomes between online and traditional sections in various 
subjects. The first line of research supported the "significant phenomenon", citing online 
learners' significant increases in learning outcomes over their traditional counterparts. The 
most widely cited literature along this line is McCollum's (1997) report. McCollum cited a 
sociology professor who divided his statistics class into two groups: one in online format and 
one in face-to-face (FtF) format. According to McCollum, online students had more online 
collaboration, and their performance outscored their traditional counterparts by an average 
of 20 percent. Later other studies also supported the "significant phenomenon" (Al-Jarf & 
Sado, 2002; Day, Raven, & Newman, 1998; Liu, 2005; Nesler, Hanner, Melburg, & McGowan, 
2001 ). 

However, the second line of research in this area supported the "no significant 
phenomenon", citing no differences in learning outcomes between online and traditional 
groups. Navarro and Shoemaker (1999) found that about 90% online learners in a graduate 
MBA class believed that they learned as much as or more than they would have in a 
traditional classroom. Later other studies also supported the "no significant phenomenon" 
(Gagne 8i Shepherd, 2001; Jones, 1999; Johnson, 2002; Johnson, Aragon, Shaik, & Palma- 
Rivas, 2000; Ryan, 2000; Schulman & Sims, 1999). 



In addition, according to recent research (Atkins 8i Vasu, 2000; Gbomita, 1997; Snider 8i 
Gershner, 1999), a teacher's attitude has a significant influence on one's computer adoption 
or implementation behavior in the classroom. A recent study by Liu and Huang (2005) found 
that in-service teachers' concerns as a whole in the United States were very intense in many 
stages of concern toward technology integration such as informational, personal, and 
refocusing stages, as measured by Hall, George, and Rutherford's (1977) Stages of Concerns 
Questionnaire (SoC). Some recent studies have investigated the effects of online instruction 
on in-service teachers' personal factors. According to Mills (1999), elementary school 
teachers' concerns and perceptions of an integrated learning system (ILS) affect the way 
they implement ILS. That is, one's attitude or concern about technology is a critical factor in 
terms of how rapidly and/or successfully one integrates technology into one's teaching. Liu, 
Lavelle, and Andris (2002) also found that participation in an online instructional technology 
course resulted in a positive modification of K-12 in-service teachers' locus of control. 
Additionally, Ertmer, Bai, Dong, Khalil, Park, and Wang (2002) found that participation in an 
online professional development course can enhance school administrators' ideas about 
technology integration and methods to support teachers' integration practices in K-12 
schools. 

A review of related literature in the area indicates that there have been studies investigating 
the impact of online instruction on in-service teachers' attitudes toward technology 
integration. Rudden and Mallery (1996) studied the effects of short term Internet instruction 
on pre-service teachers' concerns about technology integration. They found that even short 
term Internet instruction can promote pre-service teachers' attitudes in four of the seven 
areas— awareness, information, consequence, and refocusing, as measured by Hall et al. 
(1977) SoC questionnaire. Liu, Theodore, and Lavelle (2004) conducted a study involving in- 
service teachers in one online graduate course in education and reported that that there was 
a consistent and significant increase in all seven stages of the SoC instrument at the end of 
the online course, compared with those stage scores at the beginning of the online course. 
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But there have not been enough studies studying the effects of online instruction focusing 
on in-service teachers' attitudes, as measured by SoC questionnaire in simultaneous online 
and tradition sections of the same course. This study is designed to achieve this objective. 

This comparative study was designed to investigate the potential impact of online 
instruction affects K-12 in-service teachers' attitudes toward technology integration and 
learning performance during a semester-long graduate course in education. Of these two 
factors, teachers' learning performance will be assessed by students' final course grades in 
the online and traditional sections. 

Teachers' attitudes toward technology integration will be assessed by Hall et al (1977) SoC 
questionnaire, which has been widely used in this area. The detail of this instrument is 
described in a later section of this paper. In order to verify that both online and traditional 
sections are equivalent before the study, assessments of pre-course performance and 
attitudes toward technology integration were pretested and analyzed in the first week. 
Based on the above literature review, there are two major research hypotheses: 

> There will be no significant difference in learners' learning performance, as measured 
by the final course grades, between the online section and the traditional section at 
the completion of the course. 

> Learners' mean score in attitudes toward technology integration, as measured by 
SoC, in the online section would be significantly higher than that in the traditional 
section at the completion of the course. 

METHOD 

Participants 

All students who self-selected to enroll in EDUC501— Research Methods in Education in both 
online and traditional sections in the fall semester of 2003 were solicited for participation in 
this project in the first week. EDUC501 is a required core course in education at the master's 
level at a Midwestern state university in the United States. The students in this course were 
from different graduate programs in education. Fourteen students enrolled in the online 
section were encouraged to complete an online self-assessment of his/her online learning 
readiness. 

If not ready for online learning, students were encouraged to transfer to other traditional 
sections. All 14 participants in the online section were recruited to participate in the study. 

Prior to the study, it was expected that there would be a balanced number of students in 
each section. Typically, three sections, with approximately 80 students, are offered each 
semester. But due to the administrative decision, two sections were temporarily added in 
that semester. Thus in the fall semester of 2003, there were five sections of EDUC501. Only 
six students were enrolled in the lead investigator's traditional section. 

All six in the traditional section were recruited for participation in the first week, with one 
who withdrew from the class in the third week due to unexpected family issues. Thus, a 
total of 19 participants in both sections were asked to complete consent forms and 
demographic surveys. Pretests of course performance and attitudes toward technology 
integration in both sections were completed in the first week. 

A preliminary analysis indicated no significant difference existed in the pretests of course 
performance and attitudes in either section. 
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Instruments 

Students' final course grades in both online and traditional sections were based on their 
learning performance in two major assessments: knowledge and application. Both the online 
and traditional sections had the same application assessment that focused on collaborative 
learning and included a combination of essay writings, peer critiques, and a group research 
project. 

The application assessment is consistent with Wade's (1999) perspective. The perspective 
states that writing is a unique indicator of student's learning including communication 
between student and student, as well as between student and teacher. 

Knowledge assessment focused on individual learning and included; 

> chapter quizzes in the online section throughout the semester and 

> one objective final test in the traditional section in the final week. 

In the online section, each chapter quiz was administered as an individual open-book test 
without peer discussion. Each quiz consisted of 25 objectives multiple choice items 
regarding each chapter and was to be completed within 40 minutes. The quizzes in the 
online section were only available during a specific week and were graded instantly after the 
completion of each quiz. Online learners were pleased to get immediate quiz results and 
feedback. Students in the traditional section only had the final objective test in the final 
week. 

Hall et al (1977) SoC questionnaire was used to measure K-12 teachers' attitudes toward 
technology integration. The advantage of the SoC instrument is that it can measure, over 
time, a continuum of attitudes an individual may develop during a certain period time. This 
instrument assesses 7 stages of concern, which can be divided into internal and external 
concerns. The first four stages are internal including awareness, informal, personal, and 
management. The last three stages are external including consequence, collaboration, and 
refocusing. The questionnaire consists of 35 items that participants rate using an eight-point 
Likert scale that ranges from 'not true of me now' (0) to 'very true of me' (7). Participants 
choose the appropriate degree to which the included concerns are true of them. High 
numbers indicate high concern and low numbers, low concern. Five statements represent 
each of the seven stages. The 35 items appear in the instrument in a mixed order. The raw 
score for this scale is the simple sum of the responses to the five statements on that scale. 
The seven stages are distinctive but are not necessarily mutually exclusive. For educators, 
during the pre-teaching and early-teaching phases, they will be likely to have concerns 
related to self (internal), followed in the late-teaching phase with a shift in concerns that 
then focused on student learning and personal professional development (external). 



Experimental Design 

This study used a nonequivalent control group design. In both the experimental group 
(online section) and control group (traditional section), the dependent variables— scores in 
SoC instrument and learning performance in both sections were pretested and posttested. 
The independent variable was online vs. traditional instruction. A hybrid of instructional 
techniques was employed in this online section. Specifically, several major features of 
WebCT were used throughout the semester such as online writing and presentation, bulletin 
board discussion, and online testing and via regular e-mail. Constructivist learning theory 
was the major theoretical foundations for the online instruction in this course. The design 
and development of this online course was based on the ADDIE model (Analysis, Design, 
Development, Implementation, and Evaluation) proposed by Dick, Carey, and Carey (2001) 
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and the strategies discussed by Kearsley (2000).. For related details regarding the design 
and development of this online course as well as online instructional strategies in the 
course, please read the lead investigator's other recent publications (Liu, 2003a; 2003b). In 
order to reduce learner's learning anxiety and to maximize learning, one FtF orientation was 
scheduled in the first week for the online section. The other traditional section was primarily 
taught in an FtF format throughout the semester. Both sections were taught simultaneously 
by the lead investigator. To ensure that both sections as equivalent as possible, the 
instructional content and objectives were kept the same. 

Procedure 

The SoC instrument and the pre-course assessment were administered in the paper-and- 
pencil format in both sections during the first week as a pretest to measure the initial state 
of K-12 teachers' attitudes. The participants in the online section were then exposed to the 
online WebCT environment from the second week to the final week. The posttest was 
administered during the final week in both sections to measure the developmental state of 
the K-12 teachers' attitudes affected by online instruction over the semester. In addition, for 
the SoC questionnaire, raw scores for each of the seven scales described previously were 
tallied and converted to normed percentiles provided with the instrument for each of those 
seven scales. 

RESULTS AND DISCUSSION 

Pretests and posttests of attitudes and performance in both sections were coded and 
statistically analyzed using SPSS 12.0. The descriptive statistical results of pretests and 
posttest for final course grades and for SoC scores in all seven stages in raw scores and 
percentiles between experimental and control groups are shown in Table 1; their rank 
results are shown in Table 2. In addition, due to the extremely unbalanced participants in 
the experimental and control groups, the nonparametric statistic procedure, Mann-Whitney 
U test, was used to examine these differences between two groups. The Mann-Whitney U 
test results for SoC scores in all seven stages in raw scores and percentiles between 
experimental and control groups are shown in Table 3. 

The participants' final course grades in both sections included a combination of individual 
quizzes, assignments, and group projects. The grading assessment was criterion-based. The 
Mann-Whitney U test in Table 3 indicated that no significant differences were found in 
students' final course grades between the traditional and online sections at pretest (Z=- 
1.13, p= ..26) and at posttest (Z = -1.22, p= .22). Thus hypothesis 1 was supported. This 
finding is consistent with most other recent research findings referred to as "no significant 
phenomenon" (Gagne & Shepherd, 2001; Jones, 1999; Johnson, 2002; Johnson, Aragon, 
Shaik, & Palma-Rivas, 2000; Ryan, 2000; Schulman & Sims, 1999). 

However, the result in the present study is not consistent with the finding reported in some 
other studies (Al-Jarf & Sado, 2002; Day, Raven, & Newman, 1998; Liu, 2005; Nesler, 
Hanner, Melburg, & McGowan, 2001). Liu's (2005) quasi-experimental study was designed 
to compare the effects of online vs. traditional instruction on students' learning in two 
different sections (online vs. traditional section) of a graduate course in the United States. 
The two groups were enrolled in the graduate educational research course in the summer 
semester of 2003. The experimental group enrolled in the online section that received online 
instruction via WebCT and involved twenty-two graduate students, while the control group 
enrolled in the traditional section that received traditional instruction and involved twenty- 
one students. Participants in both groups were K-12 school teachers. Students in both 
sections completed the same chapter quizzes and a final test, as well as other essay 
writings, peer critiques, and group projects during the 10-week summer semester. Results in 
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that study indicated the experimental group significantly outperformed the control group on 
most quizzes and the final test. 

Regarding SoC scores, the Mann-Whitney U test in Table 3 indicated that significant 
difference only existed in the stage 2-personal stage in both raw score and percentile score 
between the online and traditional sections at pretest (Z= -2.32, p=. 02) and at posttest 
(Z=-2.13, p=. 03); no significant differences were found in other six stages in SoC at pretest 
or posttest (p> .05).. Thus hypothesis 2 was not supported in the present study. This seems 
inconsistent with findings from previous research. Liu, Theodore, and Lavelle (2004) 
conducted a study to investigate the experimental effects of online instruction in a graduate 
research methods course on K-12 teachers' concerns about technology integration. The 
concerns of twenty-three K-12 teachers regarding technology integration were measured 
using SoC both before and after completing an online course. The concerns were measured 
along seven dimensions: awareness, informal, personal, management, consequence, 
collaboration, and refocusing. Significant changes in all seven dimensions were found after 
the teachers' participation in a graduate online course. 

The above inconsistency in in-service teachers' learning performances and attitudes towards 
technology integration between traditional and online sections may be related to several 
reasons. These reasons include, but are not limited to: 

> both experimental and control groups are not balanced in terms of the number of 
participants in the present study, 

> in the traditional section, the instructor also did PowerPoint presentations; students 
also accessed /printed the instructor's chapter notes from WebCT before the class in 
the present study. 

Thus care should be taken when any generalization from the results in the present study is 
made to other environments. Further investigation of this topic is required in other control 
group environments. 

CONCLUSION 

This exploratory study revealed that no significant differences were found in most stages of 
SoC questionnaire or in in-service teachers' final course grades between traditional and 
online sections in the same course. The results indicate that online instruction can be a 
viable alternative and can be as important as traditional instruction for higher education. 
Although this study was conducted in the Unites States, its results may have implications for 
international educators and educational researchers since there is an increasing demand of 
online instruction in educational and other settings all over the world. This exploratory 
project not only has significant practical implications for education since many institutions 
are encouraged to offer online courses/ programs, but also promises contributions to the 
current literature in the area of online instruction and distance education. For example, 
embedded online courses may be used in place of more lengthy/ costly in-service training. 
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